1. Field of the Invention
The present invention relates to an image processing apparatus, a camera system, an image processing method, and a program that perform synthesis processing on plural images with different exposure times.
2. Description of the Related Art
An image sensor as a solid-state image sensing device represented by CCD and CMOS (Complementary Metal Oxide Semiconductor) performs photoelectric conversion using a photoelectric conversion device that accumulates charge in response to an amount of incident light.
Recently, cameras using image sensors such as CCDs and CMOS sensors have been widely used.
However, generally, the image sensor has a representable dynamic range (hereinafter, referred to as DR) narrower than that of a silver salt camera or the like, and, if a subject at a high contrast ratio is imaged, a problem that the high-brightness part is whiteout or the low-brightness part is blackout arises.
In order to solve the problem, a method of enlarging the apparent DR of an image to be output from the image sensor by imaging plural images with different exposure times and appropriately synthesizing them as one image is widely known those skilled in the art.
Here, as an example, a method of synthesizing two images with different exposure times will be explained.
Hereinafter, of the two images, the image with the longer exposure time will be referred to as “long accumulation image” and the image with the shorter exposure time will be referred to as “short accumulation image”, and an image obtained by digital synthesis of the long accumulation image and the short accumulation image will be referred to as “composite image”.
FIG. 1 shows an example of a graph showing relationships (hereinafter, referred to as input/output characteristics) between amounts of incident light to an image sensor and output levels with respect to a long accumulation image and a short accumulation image.
Here, 2001 shows an input/output characteristic regarding the long accumulation image and 2002 shows an input/output characteristic regarding the short accumulation image. 2003 shows a level of a component independent of the amount of incident light and an exposure time of noise contained in the output of the image sensor (hereinafter, referred to as “noise level”), and 2005 shows the amount of incident light with which the output level of the long accumulation image starts to saturate.
The long accumulation image has characteristics that the linearity of input and output is broken in a region with a predetermined amount of incident light and the saturation of the output level (=whiteout) occurs, however, the value of the ratio of the output level in the amount of incident light less than that of saturation to the noise level, the so-called S/N-ratio is high.
On the other hand, the short accumulation image has characteristics that the output level relative to the same amount of incident light is lower than that of the long accumulation image and the S/N is deteriorated, however, saturation is hard to occur even when the amount of incident light becomes larger.
FIG. 2 shows an example of a graph showing relationships (hereinafter, referred to as input/output characteristics regarding a composite image) between an amount of incident light and an output level of a composite image when a long accumulation image and a short accumulation image are synthesized and output as one composite image.
Here, 2104 shows an input/output characteristic regarding the composite image, and 2106, 2107 respectively show ranges of the amount of incident light in which the long accumulation image and the short accumulation image are selected in the composite image. 2001, 2002 are equivalent to those denoted by the same signs in FIG. 1, and their explanation will be omitted. Note that, as below, those denoted by the same signs are equivalent and their explanation will be made only once.
The synthesizing method shown in FIG. 2 selects the long accumulation image for a region with the amount of incident light less than a predetermined threshold value INP (a region shown by a range of an amount of incident light 2106) and a multiplication of the short accumulation image by an exposure ratio for a region with the amount of incident light equal to or more than the predetermined threshold value INP (a region shown by a range of an amount of incident light 2107). Thereby, the composite image is generated.
Here, the exposure ratio is a ratio of an amount of exposure of the long accumulation image to an amount of exposure of the short accumulation image, and the ratio of exposure times of the long accumulation image and the short accumulation image unless the long accumulation image and the short accumulation image are multiplied by different gains.
Further, the predetermined threshold value INP is generally set to the point slightly lower than the point at which the output level of the long accumulation image is saturated as shown in the drawing. For example, it is set to the point of about 50% to 80% of the amount of incident light with which the saturation is started.
The following other synthesizing methods are known as known technologies.
First, there is a method of employing ((short accumulation image)×(predetermined gain) (<exposure ratio)+(predetermined offset)) as an output of a region with the amount of incident light equal to or more than the predetermined threshold value INP (the long accumulation image is used for the range with the amount of incident light less than the predetermined threshold value INP).
Second, there is a method of mixing the long accumulation image and the short accumulation image near the boundary between the region using the long accumulation image and the region using the short accumulation image in the various synthesizing methods.
However, using these synthesizing methods, new problems arise.
FIG. 3 shows an example of a graph showing relationships (hereinafter, referred to as S/N characteristics of a composite image) between an amount of incident light and an S/N of a composite image when the synthesizing method shown in FIG. 2 is used.
Here, 2201 shows an S/N characteristic of the composite image, 2202 shows a point at which a region in which the long accumulation image is selected and a region in which the short accumulation image is selected are switched in the composite image, 2203 shows an S/N characteristic of the long accumulation image before synthesis, and 2204 shows an S/N characteristic of the short accumulation image before synthesis.
Here, the S/N characteristic 2203 of the long accumulation image before synthesis is depicted to the amount of incident light with which the output level of the long accumulation image is saturated.
The output level of the short accumulation image is 1/(exposure ratio) times the output level of the long accumulation image, however, the noise level differs little. Further, even when the short accumulation image is multiplied by the exposure ratio, both the output level and the noise level increase and the S/N does not change.
Accordingly, it is known that, in the region near the point 2202 at which the short accumulation image is started to be selected for the composite image, i.e., the region in which the relatively low-brightness part of the short accumulation image is used for output, the S/N extremely deteriorates. The deterioration of S/N is more significant when the exposure ratio is larger.
Further, for the purpose of effectively enlarging DR, the exposure control of the long accumulation image and the short accumulation image may be performed in the respective following manners.
The long accumulation image is obtained by exposure to light with higher brightness than average (backlight correction exposure control) for reduction of blackout.
The short accumulation image is obtained by exposure to light with lower brightness than average (excessive direct light correction exposure control) for reduction of whiteout.
Here, the exposure control refers to an operation of computing an exposure control evaluation value (hereinafter, referred to as “evaluation value”) using a predetermined method from respective images and controlling the exposure time, an aperture of the iris, an amount of PGA gain, etc. so that the evaluation value may be equal to a predetermined target level REF.
As below, for simplicity, only the control of the exposure time will be explained.
In a standard exposure control, for example, an integration of brightness values of images or an average obtained by dividing it by the number of pixels is used as the evaluation value.
On the other hand, in the backlight correction exposure control and the excessive direct light correction exposure control, for example, the evaluation value is obtained by dividing images into plural classes with respect to each brightness level and integrating brightness values with different weights for the classes.
Note that the control using the same weight for all classes is the control equivalent to the standard exposure control.
The backlight correction exposure control may be realized by setting the weights for the lower brightness classes larger and, contrary, the excessive direct light correction exposure control may be realized by setting the weights for the higher brightness classes larger.
Under the exposure control, the exposure ratio varies in the following fashions in response to the DR of scenes.
In a scene with the wider DR, the exposure ratio of the long accumulation image to the short accumulation image becomes larger.
In a scene with the narrower DR, the exposure ratio of the long accumulation image and to the short accumulation image becomes smaller. Ultimately, the ratio becomes one, that is, the exposure times of the long accumulation image and the short accumulation image become the same.
However, in the case where the longer and the shorter exposure are performed totally at twice in one field (or one frame) for the purpose of not lowering the moving picture resolution, for the short accumulation image, it may be occasionally necessary to perform exposure in a V-blanking period due to limitations of devices.
In this case, the upper limit of the exposure time (the longest exposure time) is extremely shorter than that of the long accumulation image, and, in the case where imaging is performed using a PAL method, for example, the longest exposure time of the long accumulation image is about 1/50 seconds and the longest exposure time of the short accumulation image is about 1/1000 seconds.
When the imaging scene becomes darker, through the exposure control, the exposure times of the long accumulation image and the short accumulation image eventually becomes closer to their longest exposure times, and this means that, even when the DR of the scene is narrow, the exposure ratio of about 20 times is provided with low illuminance.
That is, in the scene of narrow DR and low illuminance, there is no advantage of synthesis of the long accumulation image and the short accumulation image, and the S/N of the region in which the low-brightness part of the short accumulation image is only further deteriorated.
In view of the situation, for example, in a technology described in Patent Document 1 (JP-A-2000-69355) (hereinafter, referred to as Related Art 1), a method of reducing the noise by applying a predetermined filter to the region using the low-brightness part of the short accumulation image in the composite image is proposed.
Further, as another method of enlarging DR, there is a method of synthesizing two pixels different not in exposure time but in sensitivity, however, the same problems occur in the method of enlarging DR.
As a method of solving the problems, a technology described in Patent Document 2 (JP-A-2004-222182) (hereinafter, referred to as Related Art 2) is known.
In Related Art 2, when the exposure time becomes equal to or more than a predetermined time, the local noise in the composite image is reduced using only high-sensitive pixels.